User interface systems and methods for manipulating and viewing digital documents

ABSTRACT

Systems, including handheld computing devices that include system code stored within the memory and adapted to be executed by the processor. The system code can process an input byte stream that is representative of contents to be displayed on the touch sensitive display and can generate a content document file representative of an internal representation of the content. 
     A tool document file may also be stored in the memory and may provide an internal representation of a document providing an image that is representative of a graphical tool. Associated with a tool document can be tool code that is capable of processing the content document file to create an internal representation of the contents that presents the content in a manner that achieves a display effect associated with the tool. The device will also include parsing code that processes the content document file, the tool document file, and the processed internal representation to generate a screen document for display on the touch sensitive display in a manner that portrays the display effect.

REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.09/835,458, titled “User Interface Systems and Methods for Manipulatingand Viewing Digital Documents” and filed Apr. 16, 2001; which is acontinuation-in-part of U.S. patent application Ser. No. 09/703,502,titled “Systems and Methods for Digital Document Processing” and nowU.S. Pat. No. 7,055,095, issued May 30, 2006, both of which claimpriority to British patent application No. 0009129.8, titled “DigitalDocument Processing” and filed Apr. 14, 2000; and the contents of all ofwhich are herein incorporated by reference.

FIELD OF THE INVENTION

The systems and methods described herein relate to systems and methodsfor viewing and manipulating a display of digital documents, and moreparticularly to user interface systems and methods that allow a user tomanipulate and view a digital document presented on a display, such asthe display of a hand held electronic device, such as a computer, mobilecommunicator or phone; or a display device associated with a tactilecommander.

BACKGROUND OF THE INVENTION

Today, there is a strong effort to build mobile and handheld computingdevices that easily allow users to view documents, email, videopresentations, and other forms of content. To achieve this convergence,engineers and scientists have developed systems including the systemsdescribed in the above-referenced U.S. patent application Ser. No.09/703,502 entitled, Systems and Methods for Digital DocumentProcessing, the contents of which are hereby incorporated by reference.As described therein, digital content, whether a document, audio visualpresentation, or some other type of content, is processed by a softwaresystem operating on a handheld device, mobile device, or some otherplatform, and translated into a uniform internal representation that canbe processed and manipulated by the software system so that a display ofdifferent types of content can be generated by the system and presentedon the screen display of the respective device.

These systems, as well as other handheld and mobile computing systemssuch as the Palm Pilot, Compaq Ipaq, and mobile phones, are thereforecapable of providing a display of content to a user. However, thesehandheld and mobile systems are typically limited to simple inputdevices such as small and limited keyboards, commonly the keyboardspresent on a cellular phone, or by small touch screen systems, such asthe touch screens provided with the Palm computing device. Therefore,although these systems are capable of presenting content to a userwherein the content can be quite complex, these systems have limitedcapacity to allow a user to manipulate the display of that content, suchas by paging through the different pages of a document or selectingdifferent portions of a document. Therefore, although these handheld andportable systems may be quite useful, there are limitations to their usebased in part on the user interfaces available for supporting themanipulation and viewing of the content presented on these devices.

Consequently, there is a need in the art for systems and methods thatprovide improved user interface tools that make it more facile tomanipulate and view content presented by a handheld or portable device.

Additionally, there is a need in the art for user interface tools thatallow for the manipulation of content, when that content is separatedfrom its native application program.

SUMMARY OF THE INVENTION

The systems and methods described herein provide advanced user interfacetools that allow a user to more easily manipulate and view contentpresented on a mobile device or handheld device. In one particularembodiment, the systems and methods described herein provide a graphicaluser interface that exhibits a touch and feel user interface experience.More specifically, the systems and methods described herein includehand-held or mobile computer devices having a system for simulatingtactile control over a document which may be viewed on the device itselfor through a remote instruction or remote display on another unit. Thesesystems may include a housing which supports a processor, memory, and atouch-sensitive display (or display with remote touch-sensitivecontrol), system code stored within the memory and adapted to beexecuted by the processor. The system code may generate or provide adigital representation of a document, wherein the digital representationmay include data content and a page structure representative of the pagelayout of the document. Thus, in certain applications the rendered imagecan include the content of the document as well as the layout of thedocument, thereby providing an image of what the document physicallylooks like. The system may also include a rendering engine that mayinclude a parser and a renderer for rendering at least a portion of thepage layout of the digital representation on the touch-sensitivedisplay. A screen monitor can monitor the touch-sensitive screen fordetecting movement across a surface of the touch sensitive screen, andan interface process can process the detected movement to detect amotion representative of a command to alter the page structure of thedigital representation. A navigation module may be responsive to theinterface process and can change the rendered portion of the pagelayout. Thus, by altering the rendered portion of the page layout, thesystem allows a user to navigate through the digital representation ofthe document. Although the systems and methods of the invention willhave applicability and value when used in other applications and onother types of systems, for purposes of illustration, the invention willbe described with reference to those applications where the systemsfacilitate the navigation of documents presented on a hand-heldcomputing device.

According to one aspect, the invention relates to a computing devicehaving a display and a processor. The processor is configured to displaythe content of a document on the display. In response to a user input,the processor adjusts the graphic properties of selected portions of thedocument, thereby adjusting the prominence of the selected portionsrelative to other portions of the document. The user input may be aslider control which may be implemented e.g., on a touch screen. Theselected portions of the document may be hyperlinks or highlighted text,or they may be a combination of both. In one embodiment, the graphicproperties to be adjusted is the alpha transparency of the selectedportions, and relative prominence is adjusted by adjusting the alphatransparency of the selected portions of the document. The processordisplays the content of the document using the adjusted graphicproperties.

In another aspect, the systems and methods described herein provide,among other things, computer devices having a system for simulatingtactile control over a document. In one embodiment, these systemscomprise a processor, memory, and a display; system code stored withinthe memory and adapted to be executed by the processor, the system codeproviding a digital representation of a document including data contentand a page structure representative of a page layout of the document; arendering engine for rendering at least a portion of the page layout ofthe digital representation on the display; a screen monitor for,monitoring the screen to detect movement of an object across an, imagepresented on the display; an interface process for processing thedetected movement to detect a motion representative of a command toalter the rendered page structure of the digital representation, and anavigation module responsive to the interface process for changing therendered portion of the page layout, wherein altering the renderedportion of the page layout allows a user to navigate through the digitalrepresentation of the document.

These computer devices can include touch-sensitive displays where thescreen monitor monitors a touch-sensitive screen for detecting movementacross a surface of the touch sensitive display, as well as computerdisplays capable of depicting a cursor moving across a screen of thedisplay, and wherein the screen monitor detects movement of the cursoracross a surface of the display. The processor, memory, screen monitorand a display may be arranged as a data processing platform useful witha plurality of applications and devices including hand-held computers,telephones, mobile data terminal, a set top box, an embedded processor,a notebook computer, a computer workstation, a printer, a copier and afacsimile machine.

In certain optional embodiments, the computer device may also include avelocity detector for determining a velocity vector associated withmotion detected across the surface of the touch-sensitive display, aswell as means for applying a velocity characteristic to a documentwithin a display.

Additionally, these computer devices can have interface processes thatmake it more easy to navigate through a document or a collection ofdocuments and other content. These interface processes can include apage-flip detector for detecting a motion across the surface of thetouch-screen at a location presenting a portion of the page layoutgraphically representative of a corner of a document. The page-flipdetector can render a portion of the page layout representative of apage adjacent a currently rendered page. Similarly, the device caninclude a page curl detector for rendering a portion of the page layoutrepresentative of a portion of a page adjacent a currently renderedpage. Additionally, the interface process can include a gesturingprocess for detecting a predefined movement representative of a commandfor selecting a portion of the page layout to be rendered, or foraltering data content of the digital representation of the document.Still further interface controls include processes for controlling atransparency characteristic of a document presented on the display andfor controlling a transparency characteristic of selected portions ofthe document for adjusting visibility of the selected portions relativeto other portions of the document. Other interface processes can providetools, including tools representative of a magnifying tool, a ruler, atext entry cursor, a thumbnail navigation column, a thumbnail view oflinked content and a query tool.

In still other aspects, the invention provides computer devices, andrelated processes, having a context sensitive graphical interface tool.These devices may comprise a processor, memory, and a touch-sensitivedisplay; a content document file stored in the memory and beingrepresentative of an internal representation of the content; a tooldocument file stored in the memory and providing an internalrepresentation of a document providing an image that is representativeof the graphical interface tool; tool code capable of running of theprocessor and being associated with the tool document file and capableof processing the content document file to create an internalrepresentation of the content that when rendered presents the content ina manner that achieves a display effect associated with the tool;parsing code that processes the content document file, the tool documentfile, and the processed internal representation to generate a screendocument for display, and; interface code capable of running on theprocessor for allowing a user to arrange the image of the graphicalinterface tool into a selected contextual relationship over the renderedcontent and for directing the tool code to process a portion of thecontent document file associated with the selected position.

The contextual relationship between the graphical interface tool and therendered content can vary depending upon the application, and may forexample be selected from the group consisting of the relative positionof the graphical interface tool and the rendered content, the time atwhich the graphical interface tool acts on the rendered content, and thestate of the rendered content. These devices are flexible and may beembodied in different forms and devices, including, but not beinglimited to, a hand-held computer, a telephone, mobile data terminals, aset top box, an embedded processor, a notebook computer, a computerworkstation, a printer, a copier, and a facsimile machine, as well as incar systems, and domestic devices such as audio players, microwaves,refrigerators, and washing machines.

However, it will be understood by those of ordinary skill in the artthat these interface tools may be employed in other applicationsincluding applications wherein content is displayed on a conventionalcomputer workstation that includes typical input tools such as astandard keyboard and a mouse. Additionally, it will be understood thatthe systems and methods described herein also provide useful tools forproviding interfaces for embedded display systems, such as embeddedvisual displays employed as output devices. Examples of such embeddeddisplay systems can include cellular phones, copy machines that includea visual touch screen display that allows a user to select differentoptions for performing a copy job and which may also present images tothe user of the documents being copied. Other examples may include faxmachines wherein visual displays are provided to a user to allow a userto view a depiction of an incoming fax. Other embodiments andapplications of the user interface systems and methods described hereinwill be apparent to those of ordinary skill in the art.

More particularly, the systems and methods described herein provide userinterface tools that allow a user to manipulate content displayed on ascreen. In particular, the systems and methods described herein providesoftware systems that create an abstraction layer for information thatis to be presented on a display. This abstraction layer includes adocument object wherein a document object contains information, orcontent, that is to be displayed on a screen. In one implementation, allinformation displayed on a screen is treated as one document. Thus, atthe highest level, the entire contents of a screen is understood as onedocument object. Further to this embodiment, it will be understood thata document object may contain other document objects, each of which maycontain a subset of the content displayed to the user. Thus, at thescreen level, all information displayed will be understood as a singledocument wherein items, such as web pages, streamed video, and graphicalicons, presented on the screen are each understood separately asdocument objects contained within the high level screen document object.Therefore, all content displayed on a screen is treated abstractly as adocument, and this paradigm holds whether the content being displayed isinformation representative of a page of text or informationrepresentative of a user interface tool or window/desktop furniture.Accordingly, the user interface systems and methods described hereinprovide user interface tools and functionality for allowing a user tomanipulate document objects presented on a screen display.

Additionally, the systems and methods described herein provide, in oneembodiment, a handheld computing device that comprises a housing whichsupports a processor, memory, and a touch sensitive display. Further,the computing device may include system code stored within the memoryand adapted to be executed by the processor. The system code can processan input byte stream that is representative of content to be displayedon the touch sensitive display and can generate a content document filerepresentative of an internal representation of the content. A tooldocument file may also be stored in the memory and may provide aninternal representation of a document providing an image that isrepresentative of a graphical tool. Associated with a tool document canbe tool code that is capable of processing the content document file tocreate an internal representation of the content that presents thecontent in a manner that achieves a display effect associated with thetool. The device may also include parsing code that processes thecontent document file, the tool document file, and the processedinternal representation to generate a screen document for display on thetouch sensitive display in a manner that portrays the display effect.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing and other objects and advantages of the invention will beappreciated more fully from the following further description thereof,with reference to the accompanying drawings wherein;

FIG. 1 provides a functional block diagram of one system according tothe invention;

FIG. 2 depicts one example of a tool generated by a system such as thesystem depicted in FIG. 1;

FIG. 3 depicts a graphical user interface tool that presents a pluralityof thumbnail sketches for navigating through a document having aplurality of pages;

FIG. 4 depicts a magnifying graphical user interface tool according tothe invention which provides additional information within a magnifiedarea;

FIG. 5 depicts a semi-transparent and adaptively sizeable rulergraphical tool;

FIG. 6 depicts a transparent query marker graphical user interface tool;

FIGS. 7 a and 7 b depict a user interface mechanism for activating anddeactivating a graphical tool;

FIGS. 8 a and 8 b depict a user interface tool for visually enhancingselected portions of a displayed document;

FIG. 9 depicts a further user interface tool according to the invention;

FIGS. 10 and 11 depict a text entry tool according to the invention;

FIGS. 12 a-12 g depict a set of strokes for providing commands to ahand-held system; and

FIGS. 13A-13B depict a user interface tool for scrolling through adocument by applying a velocity characteristic to the document beingdisplayed.

DETAILED DESCRIPTION OF THE ILLUSTRATED EMBODIMENTS

The systems and methods described herein include systems and methods formanipulating and viewing documents displayed on a viewing surface suchas a computer terminal, a display screen, a printer, plotter, or anyother output device suitable for creating a visual representation ofhuman readable information. For the purpose of illustration, the systemsand methods will be described with reference to certain exemplaryembodiments, including hand held computer systems that include touchscreen displays and which are capable of displaying an integrated viewof content produced in different formats. In particular, the systems andmethods described herein include graphical user interface tools that arecapable of presenting tools that can be presented as content that willbe integrated along with other content being displayed on the screen.

FIG. 1 depicts a system 10 according to the invention. The system 10 isshown as a functional block diagram of a computer device of the typethat commonly includes a processor, a memory and a display. However, thesystem 10 may also be realized, in whole or in part, as a softwaresystem comprising system code capable of executing on a processor toconfigure the processor as a system according to the invention. Thedepicted system 10 includes a computer process 8, a plurality of sourcedocuments 11, a tool document file 30, a shape processor 22 and a videodisplay 26. The computer process 8 includes a plurality of documentagents 12, a library of generic data objects 16, an internalrepresentation file 14, a memory buffer or file 15, and aparser/renderer engine 18.

In the depicted embodiment, the display 26 can present the images of aplurality of different documents. Each of the representative outputsappearing on display 26 is termed a document, and each of the depicteddocuments can be associated with one separate application program, suchas Word, Netscape Navigator, Real Player, Adobe, Visio and other typesof applications. It will be understood that the term document as usedherein will encompass documents, streamed video, web pages, and anyother form of data that can be processed and displayed by the computerprocess 8.

The computer process 8 generates a single output display that includeswithin that display one or more of the documents. The collection ofdisplayed documents represent the content generated by the applicationprograms and this content is displayed within the program windowgenerated by the computer process 8. The program window for the computerprocess 8 may also include a set of icons representative of toolsprovided with the graphical user interface and capable of allowing auser to control the operation, in this case the display, of thedocuments appearing in the program window.

For the illustrated embodiment, the display 26 presents contentrepresentative of different data types in a single, integrated displayThis is in contrast to the conventional approach of having eachapplication program form its own display, which results in apresentation on the display device 26 that includes several programwindows, typically one for each application program. Additionally, eachdifferent type of program window would include a different set of userinterface tools for manipulating the content displayed in that window.Thus, the system depicted in FIG. 1 creates an integrated display thatcontains viewable images of documents of different types. This includesweb pages that would normally be viewed in a browser, word documentsthat would normally be viewed in a viewer or word processing document,PDF documents that would normally be viewed in a vector graphic reader,and streaming video that would normally be viewed in a video player.Thus, the depicted system 10 separates the content of these documentsfrom the underlying application program and presents them for display onthe screen 26.

To allow a user to manipulate the depicted documents, the system 10depicted in FIG. 1 provides a set of tools that can be used to navigatethrough a collection of documents, whether it is a multi-page textdocument, the pages of a web site or a series of time varying imagesthat make up a video presentation. To this end, as will be explained ingreater detail below, the system 10 creates documents that arerepresentative of tools and which may be displayed by the system 10 justas system 10 would display any other type of document. Thus the system10 of the invention has the advantage of providing a consistent userinterface, and only requiring knowledge of one set of tools fordisplaying and controlling the different documents.

As discussed above, each source document 11 is associated with adocument agent 12 that is capable of translating the incoming documentinto an internal representation of the content of that source document11. To identify the appropriate document agent 12 to process a sourcedocument 11, the system 10 of FIG. 1 includes an application dispatcher(not shown) that controls the interface between application programs andthe system 10. In one practice, an external application programminginterface (API) communicates with the application dispatcher whichpasses data, calls the appropriate document agent 12, or otherwisecarries out a request made by an application program. To select theappropriate document agent 12 for a particular source document 11, theapplication dispatcher advertises the source document 11 to all theloaded document agents 12. These document agents 12 then respond withinformation regarding their particular suitability for translating thecontent of the published source document 11. Once the document agents 12have responded, the application dispatcher selects a document agent 12and passes a pointer, such as a URI of the source document 11, to theselected document agent 12.

As shown in FIG. 1, the document agent 12 employs the library 16 ofstandard object types to generate the internal representation 14, whichdescribes the content of the source document 11 in terms of a collectionof document objects as defined in the library 16, together withparameters defining the properties of specific instances of the variousdocument objects within the document. The document object types employedin the internal representation 14 will typically include: text, bitmapgraphics and vector graphics which may or may not be animated and whichmay be two- or three-dimensional: video, audio and a variety of types ofinteractive objects such as buttons and icons. Vector graphics documentobjects may be PostScript-like paths with specified fill andtransparency. Text document objects may declare a region of stylizedtext.

Once documents are translated into an internal representation ofdocument objects, these objects are passed to the parser/renderer 18.The parser/renderer 18 generates a context-specific representation or“view” of the documents represented by the internal representation 14.The required view may be of all the documents, a whole document or ofparts of one or some of the documents. The parser/renderer 18 receivesview control inputs which define the viewing context and any relatedtemporal parameters of the specific document view which is to begenerated. For example, the system 10 may be required to generate azoomed view of part of a document, and then to pan or scroll the zoomedview to display adjacent portions of the document. The view controlinputs are interpreted by the parser/renderer 18 to determine whichparts of the internal representation are required for a particular viewand how, when and for how long the view is to be displayed.

The context-specific representation/view is expressed in terms ofprimitive figures and parameters. Optionally, there may be a feedbackpath 42 between the parser/renderer 18 and the internal representation14, e.g. for the purpose of triggering an update of the content of theinternal representation 14, such as in the case where the sourcedocument 11 represented by the internal representation 14 comprises atime varying multi-frame animation.

Each source document 11 provides a digital representation of a document,such as a text document, a spread sheet or some other document. Thedocument agent 12 creates an internal representation of that document.In one practice the created digital representation includes informationthat describes the page layout of the document, including informationabout page size, margins and other page layout information. The digitalrepresentation also includes information about the content of the sourcedocument, such as the text, figures, and other content information thatappears in the document. Processes for translating a known filestructure into another structure are known in the art, including systemsthat identify page structure and content information. Any of thesuitable techniques for performing this operation may be practicedwithout departing from the scope of the invention.

The output from the parser/renderer 18 expresses the document in termsof primitive figures. For each document object, the representation fromthe parser/renderer 18 defines the object at least in terms of aphysical, rectangular boundary box, the actual shape of the objectbounded by the boundary box, the data content of the object, and itstransparency. The shape processor 22 interprets the primitive object andconverts it into an output frame format appropriate to the target outputdevice 26; e.g. a dot-map for a printer, vector instruction set for aplotter, or bitmap for a display device. An output control input 44connects to the shape processor 22 and can deliver user interfacecontrol signals to generate an output suitable for a particular outputdevice 26. Thus, the parser/renderer 18 and the shape processor 22 canact as an engine that renders portions of the page layout and pagecontent on the display 26.

Additionally, FIG. 1 depicts a tool document file 30. The tool documentfile 30 may be a computer data file that stores informationrepresentative of an image, wherein that image may represent a tool suchas a magnifying glass, a cursor, a ruler, or any other type of tool. Forthe purpose of illustration, the system 10 depicted in FIG. 1 will nowbe described with reference to an example wherein the tool document file30 includes data that is representative of a graphical image of amagnifying glass. The magnifying glass image will be associated with afunction that allows a user to magnify the image of a document stored onthe display 26 by passing the magnifying glass over the respectiveimage. As will be described in greater detail hereafter, the magnifyingglass can include a central lens portion wherein portions of a documentthat fall under the lens of the magnifying glass appear to the user tobe magnified and therefore are presented in an enlarged format relativeto the rest of the underlying document. Although the below example willbe described primarily with reference to the magnifying glass tool, itwill be obvious to those of ordinary skill in the art that other typesof tools may be provided using the systems and methods described hereinand all such tools will be understood to fall within the scope of theinvention.

Turning to FIG. 2, the operation of the magnifying glass tool can beseen. Specifically, FIG. 2 depicts the display 26 wherein the display 26presents a screen document 42 that comprises a plurality of sub elementsincluding the document 44, the thumbnail document 46, the magnifyingglass document 48, and the lens document 50. The display 26 presents thescreen 42 as a single integrated document that contains sub documents 44through 50. The content provided for creating the screen 42 can comefrom one or a plurality of source documents 11 whose content appears asthe document 44 and thumbnail document 46. The screen document 42 alsocomprises content provided by the tool document file 30 that, in thisexample, contains data according to an internal representation dataformat wherein that data represents the image of the magnifying glass48. Additionally, the tool document file 30 may contain a portal objectthat creates a further document by processing the appropriate portion ofthe screen document 42 to present that content in an enlarged format toappear as the magnified document 50 appearing within the lens of themagnifying glass 48. Thus, the document appearing within the lens 50 isderived from the underlying document, and therefore this deriveddocument changes according to the context in which the magnifying glasstool 48 is employed. Accordingly, the specific behavior for the tool canvary depending on the context of its use. For example, a magnifyingglass tool may be associated with tool code that processes differentlythe content of a content document having map data than a contentdocument having text. For example, with a map, the magnifying glass toolmay process the associated content document to render handles within theassociated document structure that are tagged as only to be shown withina view created by a magnifying glass. Thus the derived documentpresented within the magnifying glass tool 48 can include additionalinformation, such as street names, tourist sites, public transportationlocations, notations or other information. In this operation, themagnifying glass tool responds to the context of the application, whichis the rendering of a view of a map. In other applications, where themagnifying glass tool is employed on text, the behavior of the tool mayresult in changes to the color or style of the text, or could result inthe presentation of text editing tools and user interface controls, suchas control buttons, pull down menus, annotation information, textbubbles, or other types of information.

Accordingly, the screen document 42 is an integration and aggregation ofinformation contained within a source document 11 and a tool documentfile 30. An application program associated with the tool document file30 can process the appropriate content to create the enlarged view 50.The magnifying tool 48 and the associated source code are capable ofidentifying that portion of the screen document 42 that is to bepresented in an enlarged format to create the enlarged view 50. The toolcode is further capable of processing the selected content to create theenlarged view 50 and clip that enlarged view within the lens area of themagnifying glass 48 to achieve the display affect of an enlarged regionof the screen 26. Thus, the tool document and the source document 11 arein the same internal representation, and thus can be merged into thescreen document 42, that can be rendered by the parser/renderer 18.

In one embodiment, the graphical tool 50 may be moved over the screen bydragging with a cursor, or if a touch-sensitive screen is present, bydragging a stylus or some other pointer across the screen of thedisplay. To process this movement, the display 26 may include a screenmonitoring process for monitoring the screen of the display 26 to detectmovement of a cursor, stylus or some other pointer across the images ofthe documents presented on the screen. Such screen monitoring processesare known in the art and any suitable process may be employed. Themonitor process, therefore allows a user sense of tactile control overthe visual representation of the document 44. The movements detected bythe screen monitor process may be passed to an interface process thatprocesses the detected motion to detect a motion representative of aknown command. The interface process may be a separate process or may bepart of the screen monitor process, as is common in the art. As theinterface module detects commands to move the tool 50, a navigationmodule can create input signals that direct the parser/render 18 tocreate a new display for presentation to the user, wherein the displaywill show the tool 50 repositioned as the user wanted.

Accordingly, the system depicted in FIG. 1 is capable of providing agraphical user interface tool that may be integrated into a screendisplay that represents a single document which contains a plurality ofsub documents, some of which sub documents include the graphical toolsthemselves. The power of this approach allows for the development ofnovel graphical user interface tools that allow a user to manipulate andview a document on a display and to simulate tactile control over thedepicted documents. These systems and methods are particularly wellsuited to use on hand held and mobile computing platforms wheretraditional input tools are lacking. Additional graphical user interfacetools that may be provided by the systems and methods described hereininclude the bubble thumbnail graphical tool depicted in FIG. 3.Specifically, FIG. 3 depicts the screen display 26 that includes ascreen document 52 which comprises a plurality of sub documentsincluding the document 44 and the thumbnail documents 60 through 72. Asshown in FIG. 3, the document 44 may be presented as a large documentemploying most of the viewing area of the display 26. In thisembodiment, the thumbnail documents 60 through 72 are arranged in avertical column within the screen document 52 at a position adjacent theleft side of the display 26. The thumbnail documents 60 through 72 varyin size with the largest thumbnail document 60 being centrallypositioned within the vertical array of thumbnail documents.

As further shown by FIG. 3 as documents in the vertical array increasein distance from the center document 60, the documents decrease in size.The measure of distance from the center document may be representativeof the distance in pages from the document 44, or may be representativeof some other measure of distance or difference, such as the amount oftime that has passed since the document was last viewed, the differencein alphabetical order, or some other characteristic. Thus documents 62and 68, which are adjacent the central document 60 are somewhat smallerthan the document 60. Further documents 64 and 70 which are adjacentdocuments 62 and 68, respectively, and further from document 60 aresmaller than documents 64 and 68 still. The decrease in size ofdocuments continues from documents 66 and 72, each of which is stillsmaller. The impression created by the array of thumbnail documents 60through 72 is employed to indicate that document 60, the largestdocument, is representative of the document 44 being displayed withinthe largest viewing area of the screen document 52. Documents 62 through72 get smaller in proportion to the “distance” from the current viewingpage 60. Accordingly, the vertical column of thumbnail document 60through 72 provide a navigation tool that a user can employ forselecting a document to appear within the large viewing area of thedisplay 26. Additionally, the user can select a document within thevertical array of thumbnails to choose a new document to appear withinthe viewing area. For example, in those applications where the screendisplay 26 is a touch sensitive screen display, the user may activate anew document to appear within the viewing area by touching therespective thumbnail document within the array of documents 60 through72. In those applications where the user is provided a keyboard, or amouse, the user may employ that particular input device for selectingwhich of the documents within the array of documents that the user wouldlike to appear within the viewing area. In an optional embodiment, theuser can scroll through the thumbnails to find the document of interest.Optionally, scrolling through the thumbnail documents can result in thedocument 44 changing with the scrolling of the document. Alternatively,the scrolling of the thumbnail documents can occur independently fromany changing of the document 44, with the document 44 only changing whena new thumbnail document is selected.

Accordingly, as the systems and processes described herein may employthumbnail images for queuing a user during navigation to generatethumbnail images, the systems and processes described herein can includeany suitable thumbnail generator processes including those known in theart, including those thumbnail generations that generate live, oranimated thumbnails.

FIG. 4 depicts a further embodiment of the systems and methods describedherein wherein the magnifying tool, earlier shown in FIG. 2, isassociated with tool code that results in information not earlierpresented in a document to appear within the lens area of the magnifyingglass object. More specifically, FIG. 4 depicts a display 26 thatincludes a screen document 42, which in this view appears as a map. FIG.4 further depicts the magnifying glass tool 48 that includes the lensarea 50. As shown in FIG. 4 the magnifying tool 48 is positioned over aportion of the map 42. As described above, the tool code associated withthe magnifying glass 48 is capable of presenting an enlarged view of therelevant portion of the screen document 42. As additionally shown inFIG. 4 the magnified portion 50 also includes additional information.For example, in the mapping application depicted in FIG. 4, the enlargedview 50 may include additional mapping information such as minor roads,locations of interest, or other information relevant to the contentbeing magnified. Additionally, however, the magnifying glass may beassociated with tool code that changes that color of the information, ora portion of the information, within the viewing area 50, or presentsuser interface information such as control buttons, pull down menus,annotation information, text bubbles, or other types of information.Accordingly, the specific behavior for the tool can vary depending thecontext of its use. For example, as described above the magnifying glasstool may be associated with tool code that processes differently thecontent of a content document having map data than a content documenthaving text. Thus, the systems described herein provide contextsensitive tools and processes.

FIGS. 5 and 6 depict further embodiments of graphical user interfacetools that may be provided by the systems and methods described herein.In particular FIG. 5 depicts a screen 26 that includes a screen document42 that comprises two documents 80 and 81, and a ruler 82. The twodocuments 80 and 81 are meant to represent similar types of documents,each being text documents printed on the same size paper, such as A4paper. However, as the scale of the presentation of document 81 islarger than the scale of presentation for document 80, FIG. 5 depictsthe two documents 80 and 81 as pages of text, wherein one page is largerthan the other. Thus, documents 80 and 81 are similar documents thathave been rendered with different scaling factors. As depicted in FIG. 5the ruler 82 may be a floating semi-transparent ruler that shows thescale of each document and which can adapt to the scale of theunderlying object. This is depicted by the scale of the ruler 82increasing in size as the length of the ruler travels from document 80to document 81. FIG. 5 portrays that the scale of the ruler 82 changesin proportion to the scale of the underlying documents. Thus, the ruler82 provides a context sensitive user interface tool that is capable ofadjusting the scale of the ruler in response to the presentation scaleof the content. Turning to FIG. 6, a further user interface tool, thefloating semi-transparent query mark 84 is depicted wherein the querymark 84 may display annotations for the underlying object. FIG. 6depicts that the display 26 includes a screen document 42 that comprisesa document 80 and a floating semi-transparent query tool 84. The querytool 84 when activated, either by dragging an image of the query toolonto the document, or by selecting a query tool icon already positionedover a document, will present text 88 that may include informationrepresentative of an annotation of the underlying document 80.

FIG. 7 a depicts one method for presenting to a user the available userinterface tool. In particular, FIG. 7 a depicts a screen 26 thatincludes a tool button 90. The tool button 90 provides a graphicalrepresentation of the magnifying tool 48. To activate the magnifyingtool 48 the user may click, either by use of a mouse, keypad or touchscreen, and drag from the magnifier tool button an image of the tool 48.By clicking on the tool button 90 the system will process informationfrom the tool document file to create the image of the document 48 shownin FIG. 7 a. FIG. 7 b depicts that the user interface can allow in onepractice the user to push the magnifying tool 48 off the screen 26,optionally in any direction. By pushing the tool off the screen 26 theuser removes the tool 48 and restores the icon, or the tool button 90 tothe screen.

FIGS. 8 a and 8 b depict a further tool of the type that may be employedwhen viewing documents that include links, or other types of pointers toother documents, or other content. Specifically FIG. 8 depicts a toolwherein a document 100 includes links 102 to another document. For thesedocuments, the systems and methods described herein may provide a slidercontrol 104, as well as buttons, switches, or some other control. Thedepicted slider control 104 may enhance the user interface view of thedocument 100 such that by sliding the control 104, the user can controlthe prominence of the link 102 within the document. Thus, the tool 104allows the user to adjust the prominence of links within a document suchas the document 100 so that links may be more easily identified by theuser. FIGS. 8 a and 8 b further depict that document 100 may includehighlighted text such as the depicted highlighted text 108. As with thelink 102 the slider control 104 may allow the highlighted text 108 tomaintain its transparency while the transparency of the remainingportions of document 100 vary as the slider control 104 is varied by theuser. In operation, the slider control 104 can allow the user to adjustthe transparency, or alpha figure, of the objects that make up thedocument 100 other than the objects that make up the link 102 orhighlighted text 108. However, other techniques for fading or enhancingportions of a document, may be practiced.

Turning to FIG. 9, a further graphical user interface tool is presentedwherein a document 100 includes links 102. Moreover the links 102 may beassociated with a floating thumbnail document 110. As shown in FIG. 9 acontrol 112 may be presented to the user. Upon activating the control112, those links 102 within the document 100 may be associated with afloating thumbnail representative of a page view of the page associatedwith the respective link 102. Additionally, FIG. 9 depicts that in oneoption practice the display may further include a collegian of readahead thumbnail documents 114. The thumbnail documents 114 may berepresentative of those documents that are associated with links 102within document 100, or that are associated with other pages of thedocument 100 when that document 100 is a multipage document.

FIG. 10 depicts a further example of a graphical user interface toolaccording to the invention. Specifically, FIG. 10 depicts a hand heldcomputing device 120 having a set of characters 122 appearing on itsdisplay. As further shown in FIG. 10 a cursor window 124 appears overone character within text display 122. In the depicted embodiment thecursor window 124 provides a soft, semi-transparent text entry platefloating over the current text position. The plate may move with themoving text position and/or the plate may maintain its position as thetext itself scrolls to the left to accommodate movement of text underthe plate 124. As discussed above, the text plate cursor 124 may resultfrom a tool document file 30 processed by the system 10 of FIG. 1. Thetool document file may include an internal representation of the textplate 124 as it appears on the device 120. In one embodiment, the handheld device 120 includes a touch sensitive screen that allows a user toemploy a stylus for forming characters that will appear on the screenwithin the text entry plate 124. A design and development of suchsystems that allows such text entry are well known in the art and any ofthe suitable systems may be employed with the systems and methodsdescribed herein. In operation, a user may move a stylus across thescreen of the device 120 to form letters that will appear within thetext entry plate 124. This operation is depicted in FIG. 11 wherein aseries of text entry procedures 130 through 138 are depicted.

Particularly, FIG. 11 depicts text entry step 130 wherein the cursorplate 124 appears on the display of the device 120. A user may trace aletter within space defined by the cursor 124, or in another area andoptionally, the tracings may appear within the area defined by the textentry cursor 124. The tracings entered by the user may be processed by acharacter recognition system of the type known in the art to associatethese markings with a character, such as the letter L in this example.As shown in step 132 once character recognition has been completed therecognized character L may be presented on the display and the cursormay move, or the text may scroll, but in either case the cursor 124becomes available for the user to enter more text. Thus as shown in step134 the user may enter text until a word is formed. In step 138, uponentry of a complete word the user may move the cursor 124 a space awayfrom the written word and begin again tracing characters that willappear within the text entry cursor 124. Thus, the depicted cursor 124provides a tool that allows in-line insertion of content into adocument, such as by inserting a section of text into an existing lineof text that occurs within the document. In other applications, toolscan be provided that edit images, such as by erasing content, changingcolors or performing other applications.

As described above, for those systems that include a touch sensitivedisplay the systems and methods described herein may provide forallowing a user to use a stylus to trace markings on the display andthese marking may be interpreted by the system for allowing characterentry. Additionally, FIG. 12 a through 12 b depicts a series ofrepresentative command strokes that a user may enter by moving stylusacross the touch sensitive screen. Each of the command strokes depictedin FIG. 12 a through 12 g may be associated with a user interfacecommand that the user may employ for manipulating and viewing documents.For example, FIG. 12 a depicts a stroke wherein the user forms a roundedcheck mark that the system may be associated with a command that affirmsan action proposed by the system. Similarly, FIG. 12 b depicts a strokethat forms a plurality of peaks and troughs on the display and that maybe associated with a command to delete content from a display. FIG. 12 cdepicts a circular clockwise stroke that may be associated with having adocument returned to a home page or to start over and FIG. 12 d depictsa straight line diagonal upstroke that indicates a delete, clear, or nocommand. FIG. 12 e depicts a box stroke running counterclockwise andindicating a paragraph select command and FIGS. 12 f and 12 g depictstrokes that indicate a request by the user to move to the next orrespectively previous document. It will be understood by those ofordinary skill in the art that as the systems and methods describedherein include systems and methods that work with document of differenttypes such as Word documents, web pages, streaming media, and othertypes of content the meaning of the different strokes may vary accordingto the application. For example, the circular clock wise rotation of 12c may indicate for a document that is representative of web page arequest to return to a web page associated with the web page document.Alternatively, the use of the circular clock wise stroke of 12 c whenviewing streamed media content may indicate a request to start over,causing the streamed video to stop and restart from the beginning.Accordingly, it will be apparent to those with ordinary skill in the artthat the stroke command depicted in FIGS. 12 a through 12 g may havedifferent meanings depending on their applications.

FIG. 13 a and FIG. 13 b depict a command stroke that may be employed bya user for clicking and dragging a document to cause page movement ofthat document within the viewing area. In the depicted embodiment,during a document drag operation document a velocity detector processtakes position readings periodically, such as every centi-second. Fromthese position readings a page velocity determination may be made. Thepage velocity determination may be employed for allowing the userinterface to present a more natural way of moving documents through aviewing space. To this end, a process may employ the velocitydetermination to direct the parser/render 18 to redraw the document in aseries of pictures that will portray the document as moving across thescreen. For example, a user may drag a document at a certain speed andthen release the stylus, mouse or other input device from the document.Optionally, upon release the document may stop moving. However, in analternative practice the page may continue to move in the establisheddirection until the user indicates that the document is to stop movingsuch as clicking on the document. For multi page documents the velocitymeasure may be used for panning different pages of the document acrossthe screen at a rate determined by the page velocity set when the userdrags one page of the document across the screen. Optionally, thevelocity may decrease by a constant page inertia until it reaches zerovelocity and page scrolling ceases; during page panning further velocitydetection can be used to increase (accumulate) the page velocity andhence movement against the page inertia enabling smooth continuousmovement of the page between rapid sequential drag operations.

Additionally and optionally, other user interface processes may beprovided to enhance the user experience of having tactile control overthe document. For example, the user interface may include a page-flipdetector for detecting a motion on the display 26 at a location of thedisplay associated with the upper right hand corner of document 44 inFIG. 2. If the page-flip detector, or the screen monitor, detect abrushing motion across the surface of the document 44, the page-flipdetector can direct the parser/render 18 to “flip” the page, causing thenext page, chapter, scene or other section to be displayed. Motions maybe detected in either direction for page-flipping back and forth, andthe page flip detector may be context sensitive, generating a newdisplay suited to the application and type of content. Optionally, theinterface process may include a page curl detector that can operatesimilar to the page-flip detector, except that a motion in the upperright corner of document 44 can cause the page curl detector to directthe parser/render 18 to redraw the screen 42 or document 44 so that thecorner of the document 44 is curled downward and a portion of theunderlying page is presented. Both the page-flip and page-curl detectorsmay be computer processes that can generate instructions to theparser/renderer 18 to achieve the desired effect. Additionally, apage-zoom detector (such as a double-click over the page area) can befollowed by an upward/downward movement to zoom in/out of the view. Thisfunction may be advantageously combined with the velocity detector toprovide an inertial zoom feature.

It shall be obvious to those of skill in the art that although FIG. 1graphically depicts the user interface system 10 as functional blockelements, these elements can be realized as computer programs orportions of computer programs that are capable of running on a dataprocessor platform to thereby configure the data processor as a systemaccording to the invention. Moreover, although FIG. 1 depicts the system10 as an integrated unit, it will be apparent to those of ordinary skillin the art that this is only one embodiment, and that the invention canbe embodied as a computer program distributed across multiple platforms.

As discussed above, the user interface systems described above can berealized as a software component operating on a data processing system,including hand-held computing platforms, as well as more conventionalcomputing platforms, such as a Unix workstation. In these embodiments,the user interface systems can be implemented as a C language computerprogram, or a computer program written in any high level languageincluding C++, Fortran, Java or BASIC. Additionally, in an embodimentwhere the platform is primarily a microprocessor, microcontrollers orDSPs, the user interface systems can be realized as a computer programwritten in microcode or written in a high level language and compileddown to microcode that can be executed on the platform employed. Thedevelopment of such systems is known to those of skill in the art, andsuch techniques are set forth in the literature, including for exampleDigital Signal Processing Applications with the TMS320 Family, VolumesI, II, and III, Texas Instruments (1990). Additionally, generaltechniques for high level programming are known, and set forth in, forexample, Stephen G. Kochan, Programming in C, Hayden Publishing (1983).It is noted that DSPs are particularly suited for implementing signalprocessing functions, including preprocessing functions such as imageenhancement through adjustments in contrast, edge definition andbrightness. Developing code for the DSP and microcontroller systemsfollows from principles well known in the art.

Additionally, it is to be understood that although FIG. 1 graphicallydepicts the computer process 8 as comprising a plurality of functionalblock elements, these elements can be realized as computer programs orportions of computer programs that are capable of running on the dataprocessing platform to thereby configure the data processing platform asa system according to the invention. Moreover, although FIG. 1 depictsthe system 10 as an integrated unit of a process 8 and a display device26, it will be apparent to those of ordinary skill in the art that thisis only one embodiment, and that the systems described herein can berealized through other architectures and arrangements, including systemarchitectures that separate the document processing functions and userinterface functions of the process 8 from the document display operationperformed by the display 26.

Those skilled in the art will know or be able to ascertain using no morethan routine experimentation, many equivalents to the embodiments andpractices described herein. Moreover, the systems and processes of theinvention have wide application and can be employed in a range ofdevices including hand-held computers, telephones, mobile dataterminals, set top boxes, an embedded processor, a notebook computer, acomputer workstation, a printer, a copier, facsimile machine and othersystems. Additionally, it will be understood by those of skill in theart, that the systems described herein may be practiced with anysuitable interface devices, including touch-sensitive screens and pads,mouse input devices, keyboards and keypads, joysticks, thumb wheeldevices, a mouse, a trackball, virtual reality input systems, voicecontrol systems, eye movement control systems, and any other suitabledevices. Thus, it will also be understood that the systems describedherein have many uses and provide advantages over the prior artincluding providing a set of interface processes and systems thatprovide sophisticated manipulation of different document types.

Accordingly, it will be understood that the invention is not to belimited to the embodiments disclosed herein, but is to be understoodfrom the following claims, which are to be interpreted as broadly asallowed under the law.

1-12. (canceled)
 13. A device for displaying additional contentinformation of magnified documents, comprising: a display for displayingcontent of a screen document or a zoomed document with an additionalinformation in the zoomed document as a bubble; a processor forconfiguring the display of content including a selected portion of thescreen document to generate the zoomed document; wherein the zoomeddocument is a view of contents of a portion of the screen document thatis zoomed in for enlargement.
 14. The device according to claim 13,wherein the processor uses a tool document file comprising a tool codefor processing a selected portion of the screen document appearingwithin a lens area of a magnifying tool to generate the zoomed document.15. The device according to claim 14, wherein the tool code of the tooldocument file configures the processing of text data differently thanthe processing of map data.
 16. The device according to claim 13,wherein the additional information in the zoomed document comprisesstreet names.
 17. The device according to claim 13, wherein theadditional information in the zoomed document comprises roads.
 18. Thedevice according to claim 13, wherein the additional information in thezoomed document comprises minor roads.
 19. The device according to claim13, wherein the additional information in the zoomed document comprisesat least one location of interest.
 20. The device according to claim 13,wherein the additional information in the zoomed document comprises userinterface information.
 21. The device according to claim 13, wherein theadditional information in the zoomed document comprises publictransportation locations.
 22. The device according to claim 13, whereinthe display comprises a touch screen display for detecting tactile inputand for display content of the screen document or the zoomed documentwith the additional information as a bubble in an integrated display.23. The device according to claim 22, wherein a magnifying tool moves inresponse to sensing touch by the display.
 24. The device according toclaim 22, wherein a magnifying tool moves in response the displaysensing a stylus being moved.
 25. The device according to claim 22,where a magnifying tool moves across the map of the display in responseto a pan command.
 26. The device according to claim 25, wherein the pancommand is activated by sensing inertia and/or velocity.
 27. The deviceaccording to claim 17, wherein the additional information of the roadsincludes other relevant information comprising traffic status.
 28. Amethod for displaying additional content information of magnifieddocuments, comprising: displaying together on a display content of ascreen document or a zoomed document with an additional information inthe zoomed document as a bubble; configuring by a processor a display ofcontent including a selected portion of the screen document to generatethe zoomed document; wherein the zoomed document is a view of contentsof a portion of the screen document that is zoomed in for enlargement.29. The method according to claim 28, wherein the processor uses a tooldocument file comprising a tool code for processing a selected portionof the screen document appearing within a lens area of a magnifying toolto generate the zoomed document.
 30. The method according to claim 29,wherein the tool code of the tool document file configures theprocessing of text data differently than the processing of map data. 31.The method according to claim 28, wherein the additional information inthe zoomed document comprises street names.
 32. The method according toclaim 28, wherein the additional information in the zoomed documentcomprises roads.
 33. The method according to claim 28, wherein theadditional information in the zoomed document comprises minor roads. 34.The method according to claim 28, wherein the additional information inthe zoomed document comprises at least one location of interest.
 35. Themethod according to claim 28, wherein the additional information in thezoomed document comprises user interface information.
 36. The methodaccording to claim 28, wherein the additional information in the zoomeddocument comprises public transportation locations.
 37. The methodaccording to claim 28, wherein the display comprises a touch screendisplay that detects tactile input and displays the screen document orthe zoomed document with the additional information in the zoomeddocument as a bubble.
 38. The method according to claim 37, wherein amagnifying tool moves in response to sensing touch by the display. 39.The method according to claim 38, wherein the magnifying tool moves inresponse the display sensing a stylus being moved.
 40. The methodaccording to claim 38, wherein the magnifying tool moves across the mapdata of the display in response to a pan command.
 41. The methodaccording to claim 42, wherein the pan command is activated by sensinginertia and/or velocity.
 42. The method according to claim 32, whereinthe additional information of the roads includes other relevantinformation comprising traffic status.